{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task Adherence Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates to how to use task adherence evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Task Adherence evaluator measures how well the agent adheres to their assigned tasks or predefined goal.\n",
    "\n",
    "The scoring is on a 1-5 integer scale and is as follows:\n",
    "\n",
    "  - Score 1: Fully Inadherent\n",
    "  - Score 2: Barely Adherent\n",
    "  - Score 3: Moderately Adherent\n",
    "  - Score 4: Mostly Adherent\n",
    "  - Score 5: Fully Adherent\n",
    "\n",
    "The evaluation requires the following inputs:\n",
    "\n",
    "  - Query    : The user query. Either a string with a user request or a list of messages with previous requests from the user and responses from the assistant, potentially including a system message.\n",
    "  - Response : The response to be evaluated. Either a string or a message with the response from the agent to the last user query.\n",
    "\n",
    "There is a third optional parameter:\n",
    "  - ToolDefinitions : The list of tool definitions the agent can call. This may be useful for the evaluator to better assess if the right tool was called to adhere to user intent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize Task Adherence Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation import TaskAdherenceEvaluator, AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "task_adherence_evaluator = TaskAdherenceEvaluator(model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating query and response as string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Failure example, there's only a vague adherence to the task\n",
    "result = task_adherence_evaluator(\n",
    "    query=\"What are the best practices for maintaining a healthy rose garden during the summer?\",\n",
    "    response=\"Make sure to water your roses regularly and trim them occasionally.\",\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Success example, full adherence to the task\n",
    "result = task_adherence_evaluator(\n",
    "    query=\"What are the best practices for maintaining a healthy rose garden during the summer?\",\n",
    "    response=\"For optimal summer care of your rose garden, start by watering deeply early in the morning to ensure the roots are well-hydrated without encouraging fungal growth. Apply a 2-3 inch layer of organic mulch around the base of the plants to conserve moisture and regulate soil temperature. Fertilize with a balanced rose fertilizer every 4 to 6 weeks to support healthy growth. Prune away any dead or diseased wood to promote good air circulation, and inspect regularly for pests such as aphids or spider mites, treating them promptly with an appropriate organic insecticidal soap. Finally, ensure that your roses receive at least 6 hours of direct sunlight daily for robust flowering.\",\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Query and Response as String with Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example with a single tool definition dict\n",
    "query = \"Can you recommend a science fiction book for me?\"\n",
    "response = \"I found 'Dune' by Frank Herbert for you. It's a classic science fiction novel set in a distant future.\"\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"get_book\",\n",
    "    \"description\": \"Retrieve a book recommendation for a specified genre.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\n",
    "            \"genre\": {\"type\": \"string\", \"description\": \"The genre for which a book recommendation is requested.\"}\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "result = task_adherence_evaluator(\n",
    "    query=query,\n",
    "    response=response,\n",
    "    tool_definitions=tool_definition_dict,\n",
    ")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating query and response as list of messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = [\n",
    "    {\"role\": \"system\", \"content\": \"You are an expert in literature and can provide book recommendations.\"},\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T08:00:00Z\",\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"text\", \"text\": \"I love historical fiction. Can you recommend a good book from that genre?\"}\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T08:00:05Z\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [{\"type\": \"text\", \"text\": \"Let me fetch a recommendation for historical fiction.\"}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T08:00:10Z\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"tool_call_20250314_001\",\n",
    "                \"name\": \"get_book\",\n",
    "                \"arguments\": {\"genre\": \"historical fiction\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T08:00:15Z\",\n",
    "        \"role\": \"tool\",\n",
    "        \"tool_call_id\": \"tool_call_20250314_001\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_result\",\n",
    "                \"tool_result\": '{ \"book\": { \"title\": \"The Pillars of the Earth\", \"author\": \"Ken Follett\", \"summary\": \"A captivating tale set in medieval England that weaves historical events with personal drama.\" } }',\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-14T08:00:20Z\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"Based on our records, I recommend 'The Pillars of the Earth' by Ken Follett. This novel is an excellent example of historical fiction with a rich narrative and well-developed characters. Would you like more details or another suggestion?\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"get_book\",\n",
    "        \"description\": \"Retrieve a book recommendation for a specified genre.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"genre\": {\"type\": \"string\", \"description\": \"The genre for which a book recommendation is requested.\"}\n",
    "            },\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = task_adherence_evaluator(\n",
    "    query=query,\n",
    "    response=response,\n",
    "    tool_definitions=tool_definitions,\n",
    ")\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
